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Abstract 

A synfire chain is a network that can generate repeated spike patterns with millisecond pre- 
cision. Although synfire chains with only one activity propagation mode have been intensively 
analyzed with several neuron models, those with several stable propagation modes have not been 
thoroughly investigated. By using the leaky integrate-and-fire neuron model, we constructed a 
layered associative network embedded with memory patterns. We analyzed the network dynamics 
with the Fokker-Planck equation. First, we addressed the stability of one memory pattern as a 
propagating spike volley. We showed that memory patterns propagate as pulse packets. Second, 
we investigated the activity when we activated two different memory patterns. Simultaneous ac- 
tivation of two memory patterns with the same strength led the propagating pattern to a mixed 
state. In contrast, when the activations had different strengths, the pulse packet converged to a 
two-peak state. Finally, we studied the effect of the preceding pulse packet on the following pulse 
packet. The following pulse packet was modified from its original activated memory pattern, and 
it converged to a two-peak state, mixed state or non-spike state depending on the time interval. 

PACS numbers: Valid PACS appear here 
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I. INTRODUCTION 



How our brains encode information is one of the most fascinating questions in neuro- 
science. Some researchers expect that repeated spike patterns, which are observed repeat- 
edly with millisecond precision, play a key role in the functioning of the cerebral cortex. 
Such precise patterns are observed in vivo [1, 2] and in vitro[3}. The synfire chain [4] model 
is regarded as a model for generating repeated spike patterns. A synfire chain is a function- 
ally but not anatomically feed-forward network, and it transmits synchronous spikes called 
a pulse packet. Because of the synchrony, it can reproduce spike patterns with millisecond 
precision. Synfire chains have been intensively studied theoretically, [5, 6] and have been 
confirmed to exist in vivo in an iteratively constructed network [7]. 

Many of the studies on synfire chains use a homogeneous network structure, and the 
synfire chains have only one stable propagation mode, i.e., spatially uniform synchronized 
activity. In contrast, synfire chains with several stable propagation modes have not been 
thoroughly investigated [8, 9]. In particular, activity when network is driven toward several 
propagation modes simultaneously or successively has yet to be investigated. 

In this paper, we construct a layered associative network composed of identical leaky 
integrate-and-fire neurons, and embed memory patterns into it. Assuming that the number 
of neurons is large enough, we describe the membrane potential distribution with the Fokker- 
Planck equation and analyze the network dynamics. 

Section II explains the details of our layered associative network, and §111 explains the 
Fokker-Planck method. Section IV describes the results of our analysis. In §IV A, we address 
the stability of single memory pattern propagation, and then investigate the interaction of 
two memory patterns by using simultaneous (§IVB1) or successive (§IVB2) activation. 
Section V is a summary and discussion. 

II. LAYERED ASSOCIATIVE NETWORK 

In this paper, we consider a layered associative memory network, in which a synaptic 
connection on layer / is given by 




(1) 
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FIG. 1: Schematic diagram of our layered associative network. Each circle means a neuron. The 
color shows one memory pattern. Black means ^ = +1, and white —1. Arrows show synaptic 
connections. Black lines means excitatory connections, and gray ones inhibitory. 

where the number of neurons in a layer is i,j = 1, . . . , N and the number of the memory 
patterns is ji — 1, . . . ,p. ^ represents the fith embedded memory pattern of the ith neuron 
on layer / and takes on a value of either +1 or —1 according to the probability, 

Prob^ = ±1] = \ (2) 

Figure 1 is a schematic diagram of this network. 

We use the leaky integrate-and-fire (LIF) neuron model, and the dynamics of the mem- 
brane potential v\{t) can be described as a stochastic differential equation, 

Hit) _ vl(t) - V iest I^{t) + 7 

where r is the membrane time constant, V Test is the resting potential, I is the mean of noisy 
input, 7](t) is white Gaussian noise satisfying < r](t) >= and < r](t)r](t') >= 5(t — t'), 
and D' is the amplitude of the noise. Input current l\ ,a {t) is obtained by convoluting the 
presynaptic firing l\(t) with the a function a(t) = a 2 texp(—at). 

poo 

Il' a (t)=(3 dt'a{t')l\{t-t>). (4) 
Jo 

l\{t) is derived from the sum of synaptic connections in which spikes are fired. 

^) = E^r 1 E*(*-<5:* 1 ), (5) 
3=1 k=i 

where t\ k indicates the times that the ith neuron on layer / fires. (3 is a conversion constant. 
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Membrane potential dynamics follow the spike-and-reset rule; when the membrane po- 
tential v\(t) reaches the threshold V t h, a spike is fired, and after the absolute refractoriness 
t re f, the membrane potential is reset to the resetting potential V reset . 

For the following analysis, we introduce the order parameter function m l,tl (t), namely the 
overlap, defined by 

N 

= ^£^£*(*-*U)- (6) 

i=i fc=i 

Here, the overlap means how much the firing pattern matches the /xth memory pattern on 
layer /. If neurons with memory patterns = +1 fire once, then f^° oD dtm l,fl (t) = 1. By 
using the overlap, lj(t) can be rewritten as 

m = \j^^ i '^(t)- (7) 

This means that the synaptic current to a neuron depends only on the overlap of the pre- 
ceding layer and its memory patterns. 

Throughout this paper, the parameter values are fixed as follows: iV = 1000, p = 3, 

= V reS et = HlV, V^h = 15 mV, t rei = 1 HIS, T = 10 HIS, J = 0.075 P A, C = 100 P F, 

jy = 1, a = 2 ms _1 , and (3 = 0.34 pA. 



III. FOKKER-PLANCK METHOD 



In this section, we introduce the stochastic analysis of the membrane potential. First, 
we define a vector whose elements are memory patterns of the ith neuron as £\ = 
(£j-'\£,-' 2 , • • • ,£j' p )- Each element takes on a value +1 or —1, and thus this vector has 2 P 
combinations. We can define 2 P groups according to £• values. We call each group a sublat- 
tice and we discriminate each sublattice with the vector £ = (^ 1 ,^ 2 , . . . ,£ p ). Each element 
£ M takes on ±1 values. 

Neurons belonging to the same sublattice receive the same synaptic current, because the 
synaptic current depends on only the overlaps and its memory pattern £\ (eq. (7)). The 
distribution of the membrane potential is known to evolve according to the Fokker-Planck 
equation, [6] 

-P^,t) = ~^-4(v,t) + 5(v - V ieset )2-^(t - t ref ), (8) 
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where, 



P^(v, t) is the distribution of the membrane potentials of the neurons belonging to sublattice 
and v l ^{t) = 2 P j]e(V th , t) is the firing rate. P^(v,t) and u l ^(t) satisfy the normalization 
condition, 

/V t h ftref 
dvPj:(v,t) +2- p dris l $ (t-T) = 2- p . (10) 
-oo J0 

Here we define the overlap vector, m l (t) = (m z,1 (t), m l ' 2 (t), . . . , m l ' p (t)). From eqs. (4) 
and (7), we can describe the synaptic current I l ^ a {t) by using | and m l (t) as follows: 

POD 

=p dt'a(t)I l £(t-t'), (11) 
Jo 

I\(t) = \i-m l -\t). (12) 
From eq. (6), we can describe the overlap m l, ^{t) by using firing rate v\(t) as 

= 2>- p ( 4(0 - E *4(*)) • (13) 



m 2. 



Here we can describe the network dynamics only by using macroscopic parameters P^(v,t), 
v^t), m l (t), and ij. 

The description with the Fokker-Planck method is consistent with the LIF simulation in 
the limit of the number of neurons belonging to each sublattice going to infinity, N/2 P — > oo. 
Therefore, we restrict the total number of memory patterns to p ~ 0(1). 



IV. RESULTS 



A. Activation of a single memory pattern 

Here we answer the question of whether each memory pattern synchronously propagates 
in this network as a pulse packet by using the LIF simulation and the Fokker-Planck method. 
We activate the first layer of the network. The initial condition is a stationary distribution 
for no external input, l\{t) = 0. We use the overlap as an index of how firing patterns 
match the memory pattern. For the first layer activation, we consider the virtual layer and 
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(a) m = 0.6 



(b) m = 0.4 



FIG. 2: Overlaps m '*(<) (Z = ■ ■ ■ 4) when the total volume of the overlap on the virtual layer is 
set to m = 0.6(a) or m = 0.4(b), and the standard deviation is set to a = 0.5 ms. The dashed line 
is obtained with the LIF simulation, and the solid line with the Fokker-Planck method. We define 
to, when m 0,1 (t) takes a peak value, as 3a = 1.5 ms. 

describe the overlap on the virtual layer of the first memory pattern as a Gaussian function 
with standard deviation a and total volume m. The volumes of the other memory patterns 
are set to 0; 



We calculate the input currents to neurons from eqs. (4) and (7), membrane potential 
dynamics from eq. (3), and the overlaps of the first memory pattern m l,1 (t) from eq. (6). 
Figures 2(a) and 2(b) plot the overlaps as dashed lines. Each figure shows the overlaps of 
the five layers vertically. Figure 2(a) is the case that the total volume of the overlap of 
the virtual layer is set to m — 0.6, and Fig. 2(b) is the case with m = 0.4. In Fig. 2(a) 
the temporal profile of the overlap becomes sharper as the activity propagates through the 
layers. Therefore, the memory pattern synchronously propagates as a pulse packet under 
this condition. In contrast, in Fig. 2(b), the overlap dies out as the activity propagates. 
That is, the memory pattern does not propagate under the latter condition. 

Next, we calculate the input current to the neurons belonging to each sublattice from 
eqs. (11) and (12), membrane potential distributions and firing rates from the Fokker-Planck 




(14) 
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FIG. 3: Each arrow means the change from an input overlap to an output overlap. The input 
overlaps are Gaussian functions, and the output overlaps are approximated as Gaussian functions, 
(a) LIF simulation result, (b) Fokker-Planck method result. 

equation (eq. (8)), and the overlaps of the first memory pattern m l,1 (t) from eq. (13). Fig- 
ures 2(a) and 2(b) plot the overlaps as solid lines. Since the overlaps of the //(^ l)th 
memory pattern are 0, it is enough to divide the neurons into the sublattices only with the 
first memory pattern. Therefore, we consider two distributions on each layer, P l + and P l _; 
P l + is the membrane potential distribution of the = (+1) sublattice, and P l _ is that of the 
^ l = (—1) sublattice. Figures 2(a) and 2(b) show the consistency between the results of the 
LIF simulation and those of the Fokker-Planck method. 

We choose the overlap of the virtual layer to be a Gaussian function with total volume m 
and standard deviation a. Now, we evaluate the total volume ml and the standard deviation 
a' of the overlap of the first layer by approximating the overlap with a Gaussian function 
by using the method of least squares. Figure 3 shows the normalized vector from (m, a) to 
{ml, a') in. Figure 3(a) shows the results of the LIF simulations and Fig. 3(b) shows those 
of the Fokker-Planck method. Figure 3 shows that if the initial input is strong enough and 
synchronous, the overlap reaches the attractor on the left. This attractor means that the 
spike pattern matches the memory pattern and the spikes are synchronized. It indicates 
that this network works as a synfire chain and the memory pattern propagates as a pulse 
packet. In contrast, when the initial input is too weak or too dispersed, the overlap dies out 
in the lower right region. The results of the LIF simulation (Fig. 3(a)) are consistent with 
those of the Fokker-Planck method (Fig. 3(b)). 

Hereafter we show only the results of the Fokker-Planck method, as all of the results are 
consistent with the LIF simulations. 
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FIG. 4: Schematic diagram of the activation of two memory patterns. We activate the first memory- 
pattern Tdciay after the second memory pattern. We define the time of the onset of the first memory 
pattern activation as 0. 

B. Activation of two memory patterns 

In the preceding subsection, we showed that in a layered associative network, a memory 
pattern could propagate as a spike packet. Here we focus on the dynamics when several 
memory patterns are activated. We analyze the simplest case, the activation of two memory 
patterns with an arbitrary interval Td e i ay . Figure 4 is a schematic diagram of pattern acti- 
vation. Here we consider two situations: simultaneous activation Td e i ay = and successive 
activation Td e i ay > 0. In both situations, we divide neurons into sublattices according to the 
signs of the patterns, because the overlaps of unfocused memory patterns are 0. The sub- 
lattices £ = (+1, +1), (+1, -1), (-1, +1), and (-1, -1) are described as (++), (H — ), ( — h), 
and ( ), and the membrane potential distributions are accordingly divided into the fol- 
lowing groups: P l ++ , P+-, P-+: an d P 1 —- We denote the firing rates of each sublattice as 
i/_ and v l __. 



1. Simultaneous Activation of two memory patterns 

Let us consider the case in which two memory patterns are activated simultaneously. The 
overlaps of the virtual layer are described as follows: 

expl^l^ =1,2 



m^{t) = { V 2 ^ 2 ) ^ " (15) 

/i^l,2. 
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FIG. 5: Overlaps m '^(t) when two memory patterns are activated with the same strength; mi = 
7712 = 0.5. We define ij, when m 0,1 (t) takes a peak value, as 3a 1 = 1.5 ms. 

Regarding simultaneous activation, we assume that neurons on the virtual layer fire once or 
not at all. The sum of total volumes of overlaps satisfies m 1 + m 2 < 1 because we consider 
two orthogonal memory patterns. Here, we stipulate that m 1 + m 2 = 1, a 1 = a 2 = 0.5 ms, 
and t\ = £q. We change only the ratio of m 1 to m 2 . 

First, we analyzed the simplest situation; m 1 = 1 and m 2 = 0. As we have seen before, 
the overlap of the second memory pattern was always 0, and the first memory pattern 
propagated as a pulse packet as in Fig. 2(a). 

Second, we analyzed balanced activation; m 1 = m 2 = 0.5. Figure 5 shows the temporal 
profiles of the overlaps. Throughout the observation, both overlaps m l ' 1 (t), m l ' 2 (t) have 
the same value and their total volumes were dtm^'^t) ~ 0.5. This situation seems to 
indicate that both memory patterns propagate with their intermediate levels. We call this 
state a mixed state. 

In order to elucidate which sublattices contain firing neurons, we focus on the firing rate 
of each sublattice. Figure 6 shows the firing rates v 1 ^. This figure shows that a mixed state 
is one in which spikes of £ = (++) propagate. Although the £ = (++) sublattice is not 
a memory pattern, the activity of £ = (++) propagates as a pulse packet. Therefore, a 
layered associative network is a synfire chain in which not only memory patterns but also 
their mixed states propagate as pulse packets. 

Next, we gradually increase m 1 from 0.5 to 1 while stipulating that m 1 +m 2 = 1. As long 
as m 1 is approximately smaller than 0.7, m 1,l (t) > m 2 ' l (t) early in several layers. Despite 
this, the network finally converges to the mixed state, as in Figs. 5 and 6. In contrast, when 
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FIG. 6: Firing rate Vg(t) of each sublattice when two memory patterns are simultaneously activated 
with the same strength(mi = 777-2 = 0.5). 



FIG. 7: Overlaps m l,fl (t) when the first memory pattern is more strongly activated than second 
memory pattern (mi = 0.8, 777.2 = 0.2). 

m 1 is larger than a certain threshold, m 1,l (t) has two peaks, and m 2 ' l {t) has one positive 
peak and one negative peak. Figure 7 plots the overlaps for m 1 = 0.8 and m 2 = 0.2. After 
convergence, dtm 1,l (t) ~ 1 and dtm 2,l (t) ~ 0. In this regard, this situation is 
similar to when only the /i — 1 memory pattern is activated. However, it is different in that 
the overlaps have two peaks. We call this state a two-peak state. 

Similar to Fig. 6, Fig. 8 shows the firing rate of each sublattice for the case of the two-peak 
state. Figure 8 indicates that neurons in £ = (++) and (H — ) sublattices fire at different 
timings. This difference is due to the difference in synaptic current strengths. In the first 
layer, i++(f) = \{m^\t) + m 2 ' l (t)) > I+-(t) = \(m l > l {t) - m 2 > l (t)) because m 2 > l {t) > 0. 
Therefore, neurons in the (++) sublattice receiving the larger current fire earlier than those 
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FIG. 8: Firing rate v^it) of each sublattice when the first memory pattern is more strongly activated 
than the second memory pattern; mi = 0.8, ni2 = 0.2. 

in the (H — ) sublattice. 

The timing difference does not grow after convergence in the two-peak state. To elucidate 
the reason, we rewrite as follows: 

/^ = l(m^(t)+m^(t)) 

-\(u l ++ (t) + u l + _(t)-u l _ + (t)-iy l __(t) 
+ v l ++ {t) + vl + (t) - v l + _(t) - vl_(t)) 

= l(" l ++ (t)-" l -(t)), (16) 

Similarly, 

J?- 1 = I - , (17) 

PI = \ K + W " = (18) 

^t 1 = I (u l __(t) - v l ++ (t)) = -Jj#. (19) 

Therefore, the activities of the ^ = (++), ( ) sublattices independently propagate from 

(H — ), ( — h) sublattices. Note that this independence cannot be achieved for p > 3 cases. In 
this p = 2 situation, the activity of the (++) sublattice independently propagates from that 
of (H — ). After each activity reaches a synchronous state, which is the attractor in Fig. 3, 
both activities propagate at the same speed. 

This analysis of firing rate dynamics indicates that the success or failure of propagation 
of the (H — ) sublattice activity can be represented as a two-peak state or a mixed state, 
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respectively. 

Finally, we gradually increased m 1 to 1. As m 1 became larger, the timing difference 
between the two peaks became smaller and finally vanished when m 1 — 1. Thus, we can 
consider the single pattern propagation as a special case of a two-peak state. 

In summary, simultaneous activation under the restriction of m 1 + m 2 = 1 gives rise to 
two states, a mixed state and a two-peak state. If (m 1 — m 2 \ is small enough, the network 
converges to a mixed state; otherwise it converges to a two-peak state. 

2. Successive Activation of two memory patterns 

Cateau and Fukai reported that when they activated two pulse packets successively with 
a short interval, the following pulse packet did not propagate because of hyperpolarization 
caused by the preceding pulse packet's propagation. [6] 

Let us consider the case in which two memory patterns are successively activated with a 
short interval, Td e i ay > 0. Here, because of the time interval, we do not restrict the number 
of spikes, which is in contrast to the simultaneous activation; thus m 1 +m 2 can be more than 
1. For simplicity, we assume that the overlaps of the virtual layer of both memory patterns 
take on the same values except for the timing, and that the parameters of the Gaussian 
functions describing the virtual layer overlaps are m 1 = m 2 = 0.7, and a 1 = a 2 = 0.5 ms in 
eq. (15). We activate the following /i — 1 memory pattern following the preceding /i — 2 
memory pattern (Fig. 4). 

When T delay is much larger than the membrane time constant r = 10 ms, for example 
^deiay = 50 ms, the £ = (++) and ( — h) sublattices are simultaneously activated by the 
preceding \x = 2 activation, and the £ = (++) and (H — ) sublattices are almost simulta- 
neously activated by the following /j, — 1 activation (data not shown). In this situation, 
the following memory pattern seemed to propagate normally without being affected by the 
preceding memory pattern's propagation. 

However, when the interval T delay is decreased, the spikes of neurons belonging to £ = 
(H — ) sublattice are gradually delayed compared with those of (+- 1-) neurons during the 
following pattern propagation. Figure 9 shows the firing rate of each sublattice when the 
interval T delay = 20 ms. This state is two-peaked. 

To investigate the reason for this delay, we plot the membrane potential distribution of 
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FIG. 9: Firing rate of each sublattice when we activate the first memory pattern T^eiay = 20 ms 
after the second memory pattern, tj, when m 0,1 (t) has a peak value, is defined as 3a 1 = 1.5 ms. 

the first layer of each sublattice after the propagation of the preceding memory pattern in 
Fig. 10. During propagation of the preceding memory pattern, neurons in the £ = (++) and 
( — h) sublattices fire and their membrane potential are reset to Keset (Fig- 10 dashed line). 

In contrast, neurons in the (H — ) and ( ) sublattices hyperpolarize because of inhibitory 

current (Fig. 10, solid line). Therefore when the input from the following memory pattern 
arrives, neurons in the (H — ) sublattice take more time than the ones in the (++) sublattice 
to reach the threshold V t h- This is the origin of the delay. 

In the deeper layers, a once dispersed (H — ) pulse packet develops into a synchronized 
pulse packet, and the interval between the preceding pulse packet and (H — ) pulse packet 
approaches a certain value, which is long enough for the membrane potential distribution 
of the (H — ) sublattice to relax to the stationary state. These dynamics are independent of 
the (++) sublattice dynamics. The (++) pulse packet also develops into the synchronized 
state, and the interval between the preceding pulse packet and the (++) pulse packet also 
converges to a certain interval. Therefore, the delay between (++) and (H — ) also converges 
to a certain delay period. 

If the time interval is further decreased, the activity of the (H — ) sublattice does not 
propagate after the following memory pattern activation. Figure 11 shows the firing rate 
of each sublattice when the time interval is T dc i a y = 15ms. This figure shows that only the 
activity of the (++) sublattice propagates whereas that of (H — ) dies out. This is the mixed 
state described in Fig. 6. The activity of (H — ) does not propagate because the smaller T de i ay 
is, the more the inhibitory effect remains and it is more difficult for neurons belonging to 
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FIG. 10: Membrane potential distribution of the first layer of each sublattice after propagation 
of the second memory pattern, 6.5 ms after £q. The membrane potential distribution of the £ = 
(++), ( — h) sublattice is plotted as the dashed line and that of (H — ), ( ) as the solid line. 



FIG. 11: Firing rate of each sublattice when the first memory pattern Td e i ay = 15 ms is activated 
after the second memory pattern. 

(H — ) to fire when the following memory pattern is activated. 

When the time interval is decreased even further, for example T de i ay = 8ms, even the 
activity of the (++) sublattice does not propagate (data not shown). This is because of 
hyperpolarization after spikes in the preceding activation. Therefore it is difficult even for 
neurons belonging to the (++) sublattice to fire when the first memory pattern is activated. 
This phenomenon is similar to the one reported by Cateau and Fukai [6]. 

V. SUMMARY AND DISCUSSION 

In this paper, we considered a model of a layered associative network constructed by LIF 
neurons and analyzed it with the Fokker-Planck method. We showed that mixed states, 
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not only memory patterns, propagated as pulse packets through the network. When we 
activated a memory pattern much more strongly than another memory pattern, we observed 
a characteristic phenomenon in which the overlaps have two peaks. On successive activations, 
the network converges to a two-peak state, mixed state, or non-spike state depending on the 
interval duration. 

The difference between the conventional Ising neuron network and our network is the 
stability around memory patterns. The conventional network converges to a memory state 
or a mixed state and not to a two-peaked state. The two-peaked state is a characteristic 
state for neurons which integrate inputs to fire because the neurons driven by different input 
strengths fire at different timings. In contrast, binary neurons fire simultaneously for uneven 
current magnitudes as long as the current reaches the threshold. For example, in Fig. 8, 
binary neurons belonging to £ = (++) and (H — ) fire simultaneously, and this means the 
network is in a /j, — 1 memory state. In the LIF network, the timing difference conveys some 
information about input balance and timing. 
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